27 research outputs found
Multiagent Deep Reinforcement Learning: Challenges and Directions Towards Human-Like Approaches
This paper surveys the field of multiagent deep reinforcement learning. The
combination of deep neural networks with reinforcement learning has gained
increased traction in recent years and is slowly shifting the focus from
single-agent to multiagent environments. Dealing with multiple agents is
inherently more complex as (a) the future rewards depend on the joint actions
of multiple players and (b) the computational complexity of functions
increases. We present the most common multiagent problem representations and
their main challenges, and identify five research areas that address one or
more of these challenges: centralised training and decentralised execution,
opponent modelling, communication, efficient coordination, and reward shaping.
We find that many computational studies rely on unrealistic assumptions or are
not generalisable to other settings; they struggle to overcome the curse of
dimensionality or nonstationarity. Approaches from psychology and sociology
capture promising relevant behaviours such as communication and coordination.
We suggest that, for multiagent reinforcement learning to be successful, future
research addresses these challenges with an interdisciplinary approach to open
up new possibilities for more human-oriented solutions in multiagent
reinforcement learning.Comment: 37 pages, 6 figure
Can Single Solution Optimisation Methods Be Structurally Biased?
open access articleThis paper investigates whether optimisation methods with the population made up of one solution can suffer from structural bias just like their multisolution variants. Following recent results highlighting the importance of choice of strategy for handling solutions generated outside the domain, a selection of single solution methods are considered in conjunction with several such strategies. Obtained results are tested for the presence of structural bias by means of a traditional approach from literature and a newly proposed here statistical approach. These two tests are demonstrated to be not fully consistent. All tested methods are found to be structurally biased with at least one of the tested strategies. Confirming results for multisolution methods, it is such strategy that is shown to control the emergence of structural bias in single solution methods. Some of the tested methods exhibit a kind of structural bias that has not been observed before
Modular Differential Evolution
New contributions in the field of iterative optimisation heuristics are often
made in an iterative manner. Novel algorithmic ideas are not proposed in
isolation, but usually as an extension of a preexisting algorithm. Although
these contributions are often compared to the base algorithm, it is challenging
to make fair comparisons between larger sets of algorithm variants. This
happens because even small changes in the experimental setup, parameter
settings, or implementation details can cause results to become incomparable.
Modular algorithms offer a way to overcome these challenges. By implementing
the algorithmic modifications into a common framework, many algorithm variants
can be compared, while ensuring that implementation details match in all
versions.
In this work, we propose a version of a modular framework for the popular
Differential Evolution (DE) algorithm. We show that this modular approach not
only aids in comparison, but also allows for a much more detailed exploration
of the space of possible DE variants. This is illustrated by showing that
tuning the settings of modular DE vastly outperforms a set of commonly used DE
versions which have been recreated in our framework. We then investigate these
tuned algorithms in detail, highlighting the relation between modules and
performance on particular problems
MA-VAE: Multi-head Attention-based Variational Autoencoder Approach for Anomaly Detection in Multivariate Time-series Applied to Automotive Endurance Powertrain Testing
A clear need for automatic anomaly detection applied to automotive testing
has emerged as more and more attention is paid to the data recorded and manual
evaluation by humans reaches its capacity. Such real-world data is massive,
diverse, multivariate and temporal in nature, therefore requiring modelling of
the testee behaviour. We propose a variational autoencoder with multi-head
attention (MA-VAE), which, when trained on unlabelled data, not only provides
very few false positives but also manages to detect the majority of the
anomalies presented. In addition to that, the approach offers a novel way to
avoid the bypass phenomenon, an undesirable behaviour investigated in
literature. Lastly, the approach also introduces a new method to remap
individual windows to a continuous time series. The results are presented in
the context of a real-world industrial data set and several experiments are
undertaken to further investigate certain aspects of the proposed model. When
configured properly, it is 9% of the time wrong when an anomaly is flagged and
discovers 67% of the anomalies present. Also, MA-VAE has the potential to
perform well with only a fraction of the training and validation subset,
however, to extract it, a more sophisticated threshold estimation method is
required.Comment: Accepted in NCTA202
Analysis of modular CMA-ES on strict box-constrained problems in the SBOX-COST benchmarking suite
Box-constraints limit the domain of decision variables and are common in
real-world optimization problems, for example, due to physical, natural or
spatial limitations. Consequently, solutions violating a box-constraint may not
be evaluable. This assumption is often ignored in the literature, e.g.,
existing benchmark suites, such as COCO/BBOB, allow the optimizer to evaluate
infeasible solutions. This paper presents an initial study on the
strict-box-constrained benchmarking suite (SBOX-COST), which is a variant of
the well-known BBOB benchmark suite that enforces box-constraints by returning
an invalid evaluation value for infeasible solutions. Specifically, we want to
understand the performance difference between BBOB and SBOX-COST as a function
of two initialization methods and six constraint-handling strategies all tested
with modular CMA-ES. We find that, contrary to what may be expected, handling
box-constraints by saturation is not always better than not handling them at
all. However, across all BBOB functions, saturation is better than not
handling, and the difference increases with the number of dimensions. Strictly
enforcing box-constraints also has a clear negative effect on the performance
of classical CMA-ES (with uniform random initialization and no constraint
handling), especially as problem dimensionality increases
Multi-surrogate Assisted Efficient Global Optimization for Discrete Problems
Decades of progress in simulation-based surrogate-assisted optimization and
unprecedented growth in computational power have enabled researchers and
practitioners to optimize previously intractable complex engineering problems.
This paper investigates the possible benefit of a concurrent utilization of
multiple simulation-based surrogate models to solve complex discrete
optimization problems. To fulfill this, the so-called Self-Adaptive
Multi-surrogate Assisted Efficient Global Optimization algorithm (SAMA-DiEGO),
which features a two-stage online model management strategy, is proposed and
further benchmarked on fifteen binary-encoded combinatorial and fifteen ordinal
problems against several state-of-the-art non-surrogate or single surrogate
assisted optimization algorithms. Our findings indicate that SAMA-DiEGO can
rapidly converge to better solutions on a majority of the test problems, which
shows the feasibility and advantage of using multiple surrogate models in
optimizing discrete problems
When to be Discrete: Analyzing Algorithm Performance on Discretized Continuous Problems
The domain of an optimization problem is seen as one of its most important
characteristics. In particular, the distinction between continuous and discrete
optimization is rather impactful. Based on this, the optimizing algorithm,
analyzing method, and more are specified. However, in practice, no problem is
ever truly continuous. Whether this is caused by computing limits or more
tangible properties of the problem, most variables have a finite resolution.
In this work, we use the notion of the resolution of continuous variables to
discretize problems from the continuous domain. We explore how the resolution
impacts the performance of continuous optimization algorithms. Through a
mapping to integer space, we are able to compare these continuous optimizers to
discrete algorithms on the exact same problems. We show that the standard
-CMA-ES fails when discretization is added to the problem
Challenges of ELA-guided Function Evolution using Genetic Programming
Within the optimization community, the question of how to generate new
optimization problems has been gaining traction in recent years. Within topics
such as instance space analysis (ISA), the generation of new problems can
provide new benchmarks which are not yet explored in existing research. Beyond
that, this function generation can also be exploited for solving complex
real-world optimization problems. By generating functions with similar
properties to the target problem, we can create a robust test set for algorithm
selection and configuration.
However, the generation of functions with specific target properties remains
challenging. While features exist to capture low-level landscape properties,
they might not always capture the intended high-level features. We show that a
genetic programming (GP) approach guided by these exploratory landscape
analysis (ELA) properties is not always able to find satisfying functions. Our
results suggest that careful considerations of the weighting of landscape
properties, as well as the distance measure used, might be required to evolve
functions that are sufficiently representative to the target landscape
Is there Anisotropy in Structural Bias?
The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Structural Bias (SB) is an important type of algorithmic deficiency within iterative optimisation heuristics. However, methods for detecting structural bias have not yet fully matured, and recent studies have uncovered many interesting questions. One of these is the question of how structural bias can be related to anisotropy. Intuitively, an algorithm that is not isotropic would be considered structurally biased. However, there have been cases where algorithms appear to only show SB in some dimensions. As such, we investigate whether these algorithms actually exhibit anisotropy, and how this impacts the detection of SB. We find that anisotropy is very rare, and even in cases where it is present, there are clear tests for SB which do not rely on any assumptions of isotropy, so we can safely expand the suite of SB tests to encompass these kinds of deficiencies not found by the original tests.
We propose several additional testing procedures for SB detection and aim to motivate further research into the creation of a robust portfolio of tests. This is crucial since no single test will be able to work effectively with all types of SB we identify